Given the continually increased reliance on computers in contemporary society, computer technology has had to advance on many fronts to keep up with both increased performance demands, as well as the increasingly more significant positions of trust being placed with computers. In particular, computers are increasingly used in high performance and mission critical applications where considerable processing must be performed on a constant basis, and where any periods of downtime are simply unacceptable.
Increases in performance often require the use of increasingly faster and more complex hardware components. Furthermore, in many applications, multiple hardware components, such as processors and peripheral components such as storage devices, network connections, etc., are operated in parallel to increase overall system performance.
One particular area to which development efforts have been directed is that of managing the peripheral hardware components utilized by a computer, e.g., storage devices, network connections, workstations, and the adapters, controllers and other interconnection hardware devices utilized to connect such components to the central processing units of the computer. Peripheral components, which are referred to hereinafter as input/output (IO) resources, are typically coupled to a computer via one or more intermediate interconnection hardware devices components that form a “fabric” through which communications between the central processing units and the IO resources are passed.
In lower performance computer designs, e.g., single user computers such as desktop computers, laptop computers, and the like, the IO fabric used in such designs may require only a relatively simple design, e.g., using an IO chipset that supports a few interconnection technologies such as Integrated Drive Electronics (IDE), Peripheral Component Interconnect (PCI) or Universal Serial Bus (USB). In higher performance computer designs, on the other hand, the IO requirements may be such that a complex configuration of interconnection hardware devices is required to handle all of necessary communications needs for such designs. In some instances, the communications needs may be great enough to require the use of one or more additional enclosures that are separate from, and coupled to, the enclosure within which the central processing units of a computer are housed.
Often, in more complex designs, peripheral components such as IO adapters are disposed on printed circuit boards, or cards, which are mounted and coupled to an IO fabric using “slots” that are arrayed in either or both of a main enclosure or an auxiliary enclosure of a computer. Other components may be mounted or coupled to an IO fabric in other manners, e.g., via cables and other types of connectors, however, often these other types of connections are referred to as “slots” for the sake of convenience. Irrespective of the type of connection used, an IO slot therefore represents a connection point, or IO endpoint, for an IO resource to communicate with a computer via an IO fabric. In some instances, the term “IO slot” is also used to refer to the actual peripheral hardware component mounted to a particular connection point in an IO fabric, and in this regard, an IO slot, or the IO resource coupled thereto, may also be referred to hereinafter as an endpoint IO resource.
Keeping in mind the goal of minimizing system downtime, many IO fabrics also support the ability to “hot plug” IO endpoint resources, such that IO endpoint resources may be disconnected from or connected to the IO fabric without having to shut down the system. Often, complex systems support the ability to dynamically reconfigure the IO fabric to accommodate the removal/addition of IO endpoint resources during runtime.
It is common in many computer systems, particularly those employing PCI-compatible IO buses and adapters, for computer programs such as device drivers to use memory-mapped input/output (MMIO) to communicate with IO endpoint resources. This allows computer programs such as device drivers executing on the computer systems to treat IO resources such as adapters as if they were logically connected directly to the processor/memory bus of the system and just an extension of the system memory occupying a particular memory address range. A device driver may then communicate with an adapter using processor load or store instructions targeting “memory” addresses that correlate directly to internal adapter facilities. In such a model the device drivers are largely unaware of the composition and arrangement of IO fabric elements, and rely on the IO fabric and IO adapters to behave as if the device drivers were simply accessing a memory region in response to a memory-mapped load or store.
The allocation of memory address ranges to IO resources such as IO adapters, however, can be problematic, particularly in complex computer systems supporting dynamic reconfiguration of IO resources during system runtime. In less complex computers such as single-user computers and the like, system firmware typically chooses a memory range for each IO slot during startup, and assigns a memory range to a particular slot only when an IO adapter is actually detected in that slot and the actual memory requirement of that particular adapter model can be determined.
An assignment of memory only to detected IO adapters at startup, however, has been found to have significant drawbacks in computers in which reconfiguring an IO fabric on an active system may be necessary. If, for example, an IO adapter is initially configured in a slot that requires 16 MB of memory, and the firmware assigns exactly that much, if the adapter is later replaced by a different IO adapter that requires more memory, the firmware must either find an unused range of memory space that can accommodate the size required by the new IO adapter or reboot the computer in order to reassign memory space to all configured IO adapters.
High performance multi-user computers, on the other hand, often require the ability to concurrently replace or add IO adapters with minimal or no impact on system availability. It is often not practical such computers to reboot the whole platform in order to “re-size” memory as a result of a change in IO adapter configuration. Therefore, the platform firmware typically must choose memory assignments throughout the IO fabric prior to knowing the memory requirements of each installed IO adapter. This becomes more compelling with IO fabrics that offer a multiplicity of IO endpoints or slots that are hot pluggable such that IO adapters may be added as the customer requires them.
Hot pluggable and on demand implies that an IO adapter can be added non-disruptively. To accomplish this, as the specific memory requirements of an IO adapter cannot be known if the IO slot is empty during platform boot, or the IO adapter is de-configured, firmware must apply some method of determining an “appropriate” amount of PCI memory for each IO slot prior to knowing the actual requirements of each possible IO adapter.
In addition, in some computer designs, the amount of memory address space available to allocate to IO resources is limited in nature. As such, it is often not an acceptable solution to simply allocate more of the memory address space to each IO slot from the outset, and irrespective of the particular IO adapter that is or will be installed in that slot.
Some designs have attempted to allocate memory address space on the basis of connector type. For example, in some PCI-based environments, IO slots and adapters may support either 32-bit or 64-bit connectors. In such designs, IO slots may be assigned different amounts of memory address space to each IO slot based simply upon connector type (e.g., 128 MB for a 32-bit connector, and 256 MB for a 64-bit connector). These choices are based on a loose principal that 32-bit adapters have less function and complexity than 64-bit adapters, and that these values (128 MB and 256 MB) represent a “good guess” at how much of the memory address space “most” IO adapters require.
However, it has become apparent from such “good guess” choices of a priori assignments that some adapters, even though they may be only a few select models, may not work at all on a particular platform as a result of these fixed allocations of the memory address space. As adapters have grown in complexity, it is not unreasonable to expect that some adapters will exceed these “good guess” values. On the other hand, while selecting higher allocations for different connector types may enable a platform to accommodate additional types of IO adapters, such higher allocations will increase inefficiency for those adapters that do not require as much memory address space, and may limit the number of adapters that may coexist in a fixed memory space.
Additionally, some systems support the use of bridge adapters that can expand an IO fabric into external enclosures to accommodate additional IO endpoints and adapters. Bridge adapters themselves, however, are typically incapable of reporting the memory requirements of the IO adapters being managed thereby. Therefore, it is possible that, while no individual IO adapter that might be plugged into an IO slot requires more than the “good guess” choice for that slot, the placement of such an “expansion” bridge adapter in that slot could result in that slot requiring a memory address space aggregating over multiple IO adapters below that bridge adapter so as to greatly exceed the “good guess” choice.
Therefore, a significant need has arisen in the art for a more efficient and flexible manner of allocating memory address space to IO resources in a memory mapped IO fabric.